Platform MultiMod: Multimodal Acquisition Data Platform

We have set up an acquisition hardware platform to acquire multimodal data in speech communication context. The system was previously composed of the articulograph Carstens AG501 (which was acquired as part of the EQUIPEX ORTOLANG), 4 Vicon cameras (a motion capture system), and an Intel RealSense camera which contains four components: a video camera, an infrared laser projector, an infrared camera, and a microphone array. With such heterogeneous hardware the synchronization is essential; this is achieved through a trigger device. All the data processing is performed with the PLAVIS software.

This year, we have replaced the 4 Vicon cameras by 8 optitrack cameras. The new motion capture system allows acquiring higher spatial and temporal resolution data, and allows faster acquisition and processing.

We are currently using the system to acquire expressive audiovisual data to build an expressive audiovisual speech synthesis in addition to a lipsync system.